Two staging clusters, replaced by a $400 cap.

Paying for staging at 3:00 AM

Oxbow runs payroll for independent restaurants: hours, tips, tax filings, and the direct deposit that has to land on Friday. Its 64 engineers in Boise ship Go services to GCP with Helm, and for three years they tested them on two staging clusters, because one was never enough for the payroll team and the tips team at the same time.

Both clusters ran all day, every day, and together they cost $2,140 a month. When Sam Rivera took over the platform team he pulled the request logs. The clusters had served traffic during 14% of the hours they were up.

"We were paying for staging at 3:00 AM on a Sunday," Rivera says. "Nobody at Oxbow is testing payroll at 3:00 AM on a Sunday."

They had also drifted apart. The second cluster carried a feature-flag setting nobody remembered making, and "it works on b" had become a thing people said in review.

What they changed

Tovah Lindqvist, a platform engineer, did the migration in the first two weeks of February. The Helm chart already existed, so preview.from pointed at it, and the runner went into Oxbow's own GCP project. Employee names, Social Security numbers and bank account numbers are masked there, before a preview reads the nightly snapshot.

preview:
  from: charts/oxbow
  sleep_after: 30m
  teardown: on_merge
  budget:
    monthly_usd: 400

The $400 was a guess: about a fifth of the old bill, which felt like a number finance would notice. The first week tested it. An engineer had pointed an uptime monitor at a preview link to watch a long import, and forgot about it. The preview stayed awake for four days. It had cost about $18 by the time Lindqvist saw it on the spend page, which she points out is the reason to have a spend page. The monitor now watches production, where it belongs.

For a month the old clusters stayed up as a fallback. Nobody deployed to either of them after the second week, and both were deleted on March 9.

What happened

• Oxbow's bill has averaged $312 a month since, against $2,140 for the two clusters. The highest month, $371, was the one with the tip-pooling rewrite.
• Previews spend 86% of their time asleep. Engineers stopped noticing the wake months ago.
• Median teardown is 2m 05s after merge, including the database volume.

The drift is gone as well, because there is nothing left to drift. Every preview is built from the chart on its own branch and seeded from last night's snapshot, so a setting that is not in the pull request is not in the preview.

Rivera is careful about what the cap does and does not do. "The cap is not why the bill is low," he says. "The bill is low because nothing runs when nobody is looking. The cap is how I explained it to finance." Oxbow moved to an annual commitment in August. The rates are on the pricing page, and the Helm setup Lindqvist followed is in the docs.